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n the quest to address income inequality and intergenerational 

poverty, it is critical to seek answers to the questions “What works?” 

and “For whom?” Education is a prime area in which to seek solutions, 

as one’s level of education completed is positively correlated with 
income later in life. This connection contributes to a specific focus on 
schools to reduce inequality and a reliance on test scores as both a measure 
of intervention effectiveness and a proximal measure of young people’s 
future outcomes. 


As Richard Murnane points out in a new essay, however, some efforts to 
improve and expand instruction have yielded improvements in test scores, 
but a narrow focus on this measure alone obscures a larger picture (2021). 
Murnane goes on to detail three interventions—including two housing 
mobility interventions and a school choice program—that did not improve 
students’ reading and math scores but did dramatically improve their 

life chances. These findings suggest what many educators and parents 
know: Larger contexts affect school performance. In the United States, 

for instance, racism is pervasive and continues to have implications for 
African Americans in all areas of their lives, including health (Kwate et al., 
2008; William & Williams-Morris, 2000), income (Rothstein, 2018), and 
wealth (Akbar et al., 2019). To address income inequality and intergen- 
erational poverty, then, education is just one instrument—we need a tool 
chest. 


Until researchers and funders recognize and attempt to account for 

the broader context surrounding the studies we conduct, our work will 
continue to shine a narrow light on potential solutions. Both research and 
data reflect the perspectives of the researcher and the data collector, as 
well as the societal context. Given the widespread fallacy that research and 
data are objective, this recognition will require intentional effort on the 
part of researchers. Ignoring these facts will only perpetuate reeommen- 
dations for incomplete policy actions. 
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Theories of Change Account for Individual Action, but not Racism 


overty is associated with poor outcomes in many areas of life 

(Lin & Harris, 2009). While there is evidence that many spells of 

poverty are relatively brief (Bane & Ellwood, 1986; Stevens, 1999), 

long spells make up the vast majority for those in poverty at any 
single point in time. And African Americans have a higher incidence of long 
spells than other groups (Hoynes et al., 2006). 


Poverty and racism are synergistic. Here, racism is defined as the organized 
system that leads to subjugating certain groups relative to others (William 
& Williams-Morris, 2000). That is, the definition of racism that I’m 
employing is synonymous with a racialized system—or structural racism— 
which does not require a psychological phenomenon of racist beliefs held 
by individuals (Bonilla-Silva, 1997). In a racialized system, the concepts 

of economics, politics, societal expectations, and ideology are all partially 
structured by the placement of people in racial categories. This placement 
may be reinforced by policy that reflects beliefs of inherent deficits or 
other internal shortcomings. This view of racism as structural is a critical 
distinction because it dramatically shifts the theory of change of inter- 
ventions that seek to improve outcomes among BIPOC with low-income 
from the individual to the larger context. 


For example, the theory of change of Moving to Opportunity (MTO), 

one of the studies cited by Murnane, is that living in high poverty areas 
diminishes the life chances of low-income residents (Goering et al., 1999), 
but vouchers to help families move to lower poverty areas can improve 
familial outcomes related to income, health, and education. While not 
based on MTO’s specific theory of change, the Chicago Housing Authority’s 
demolition of high-rise public housing buildings beginning in the 1990s 
represented a natural experiment that could be used to test its validity. 
Similarly, the theory of change underlying the Charlotte-Mecklenberg 
school choice voucher program involves a relocation of students from 
neighborhood schools (assumed to be low-performing, underfunded, and 
located in poverty-affected neighborhoods) to better educational opportu- 
nities (Deming, 2011). While the theory of change in all instances involves 
a movement to secure better resources, it does not account for the larger 
system that contributes to the inequities in the first place. In this way, 

it implicitly assumes that, with the move, individuals have the means to 
improve their children’s outcomes (and in the case of the housing inter- 
ventions, also improve their economic outcomes as adults). 
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Racism is Pervasive 


tis easy (and perhaps comforting) to regard having low income as a 

result of individual behaviors and decisions. This narrows the focus of 

attention to helping the individual make better choices. However, what 

happens when the individual’s choices are limited in numerous ways 
(subtle and not so subtle) that are outside of the control of most? The result 
is often that the larger society continues to emphasize the Horatio Alger 
ideal of pulling one’s self up despite one’s personal circumstances. While 
there are many BIPOC who have come from extremely challenging circum- 
stances to better their life chances and those of their families, overlooking 
history and maintaining silence toward barriers to success seems like an 
additional assault on BIPOC and areinforcement of a deficit perspective. 
This is particularly true for African Americans. 


For example, even after the abolition of slavery, African Americans have 
faced structural racism. This has played out, in part, in housing segre- 
gation—a nationwide phenomenon that could only be possible with the 
cooperation of the real estate industry, banking institutions, federal 
housing policy, and support for neighborhood covenants (Massey & 
Denton, 1993). Supporting public schools through funding formulas that 
rely on property taxes only reinforces institutionalized racism as unequal 
funding models give way to separate and unequal education. Add to this the 
history of employment limitations, such as historic union policies to limit 
membership by race and to use African Americans only as strikebreakers 
(Massey & Denton, 1993), coupled with the movement of high-wage jobs 
from urban industrial centers to the suburbs, high unemployment rates, 
and mass incarceration of African American males (contributing to 

single, female-headed households), and it’s not surprising that test scores 
between African American children in low-income schools lag behind 
those of their White counterparts. In fact, to compare test scores without 
acknowledging the costs that are born unevenly by African Americans who 
make decisions within the context of being Black in America (which can 
result in constrained personal decision making) seems to suggest that indi- 
vidual or family decisions are the cause of disparate outcomes. 


The unevenness of historical policy, however, is a causal mechanism that is 
often unacknowledged (Rothstein, 2018). Differences in outcomes reflect 
these circumstances, as well as individual decision making (Mullainathan 
& Shafir, 2013). As the popular YouTube video “Life of Privilege Explained 
in a $100 Race” suggests (Youngsterdam Dynamo, 2019), it is a fallacy to 
believe that everyone begins at the same starting point and has an equal 
opportunity for success when African Americans experience multiple dis- 
advantages relative to their White peers. 
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While all of this history may seem tangential to the economics of edu- 
cation, these contextual factors make it clear that education is just one 
tool to address inequality and intergenerational poverty. In the context 

of historical (and current) barriers, education may be a meager tool for 
large-scale societal change (Berliner, 2013). It is in this context that a 
reduction in test scores gaps needs to be viewed. Without acknowledging 
that the gaps themselves are symptomatic of a larger system (Billings- 
Ladson, 2006), intervention development and policymaking will continue 
to operate with the false hope of finding a silver bullet solution. 


Addressing Root Causes Matters 


he three interventions that Murnane highlights are noteworthy 
precisely because they address housing segregation, a root cause 
of educational differences (Massey & Denton, 1993; Clark, 1965). 
While addressing this issue is promising, it is not enough. Is the 
policy implication to “move folks out of these terrible neighborhoods”? 
This is ineffective given present-day housing discrimination (Christensen 
& Timmins, 2019), racial fear tactics (Karni, Haberman, & Ember, 2020; 
Gomer & Petrella, 2017), and White Americans’ growing concern of reverse 
discrimination (Earl & Hodson, 2019). But it is also problematic because 
it presumes that nothing works in high-poverty neighborhoods—a deficit 
view that focuses on lack. More fruitful avenues may include recom- 
mending mixed-income housing for all new developments, policies to 
ensure that gentrification does not result in displacement, or a dismantling 
of housing policies that have disparate impact. However, the acknowl- 
edgement of history is again needed to prevent the implementation of 
policies that may devolve into other forms of racism. A poignant cautionary 
tale is the integration management programs of the 1980s—policies geared 
to reduce racial segregation through ensuring a minority of Blacks in areas 
through quotas—which were used to enforce racism in a different manner. 
Under these programs, units set aside for Blacks typically had long wait 
lists and were smaller in number, while those for Whites were readily 
available (Lake & Winslow, 1981, cited in Massey & Denton, 1993). 


Policy recommendations can be made more actionable by taking a cul- 
turally responsive and equitable evaluation lens to the research in 

order to determine additional solutions given the evidence around the 
mechanism (Thomas et al., 2018; Stanfield, 1999). An evaluation is cul- 
turally responsive if it considers the behavior, values, customs, common 
beliefs, and context of participants and the program being evaluated (Hood, 
Hopson, & Kirkhart, 2020; Frierson et al., 2010). Equitable evaluation 
encompasses the awareness that linguistic, historical, and socioeconomic 
differences, along with structural factors, perpetuate inequity in America 
(Public Policy Associates, 2015). Evaluators can start to address these 
factors through developing research questions and protocols to account for 
these factors and use the information to push deeper understanding. 
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Solutions that address root causes—not just symptoms—will require an 
interdisciplinary approach that seeks to understand and address multiple 
areas that contribute to the problem of racial disparities (Billings-Ladson, 
2006). This is difficult work, especially given that our most rigorous 
methodology for assessing causality—random assignment—is not helpful 
in discerning the impact of a system. Consider that in the MTO study, 
there is strong internal validity about the outcomes on children asso- 
ciated with the moves to lower poverty neighborhoods, but the history of 
racialized systems in each of the cities is contextual. The Chicago Housing 
Authority, for example, is infamous for the Gautreaux class action lawsuit 
that charged the CHA and the U.S. Department of Housing and Urban 
Development with discrimination in the location of federal housing and 
the assignment of tenants. The case took 15 years for resolution through 

a verdict by the Supreme Court and yet Chicago remains one of the 

most racially segregated cities in America. This context is important to 
understand the outcomes of MTO. 


This Work is Hard, but it is the Right Thing to do 


adly, none of these points are new. Kenneth Clark addressed 

them extensively in the 1950s and 1960s, and evaluators have 

been making the call for more nuanced evaluation efforts for 

more than 40 years (see, e.g., Thomas et al., 2018; Thomas & 
Parsons, 2017; Gadson-Billings, 2006; Hood, 2001; Massey & Denton, 1993; 
and Clark, 1965). Perhaps we have reached a critical juncture in the U.S. 
where we are able to recognize the injustice that befalls some Americans 
and we are able to see the benefits our system bestows on others. Given the 
consequences of reinforcing the status quo, this growing awareness must 
encourage us to pursue our evaluation work differently. 


Iam aware that what Iam recommending cannot be done within one dis- 
ciplinary approach. “How” and “why” questions will require an interdis- 
ciplinary examination. I also acknowledge that taking this approach may 
be risky for the careers of some researchers who may not have the support 
of their colleagues, peers, or institutions. Further, I recognize that some 
funders—including large federal contracting agencies—will not want to 
support work that delves into context which seems out of scope. 


There are no clear and easy solutions, but there are starting points. 
Researchers and evaluators alike can begin in four broad areas: 
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Familiarization with the history and context of the issue(s) studied 


There is a history behind every policy and program and, given American 
history, there is a high likelihood that the history reflects the majority 

in power (Clark, 1965). As a result, it becomes necessary to understand 
whether the policy reinforces negative, stereotypic views (House, 2017) 
and to learn about the perspectives of those affected by policies and 
programs who lack the power of policy creation. That is, rather than take 
an agnostic position of looking at an issue in only its current framing, learn 
about what has been done before (and why) from multiple perspectives, 
paying special attention to unintended consequences (Richburg-Hayes, 
2014) as well as long-term outcomes (House, 2017). This supplemental 
knowledge may temper naive policy recommendations and mitigate 
implicit blame of the victim when context matters (Ross & Nisbett, 2011). 
Maria Cancian, immediate past president of the Association for Public 
Policy Analysis and Management, illustrated the value of this approach in 
understanding child support policy. While research has noted disparate 
child support remittance rates between Black and White fathers, the 
research almost uniformly ignored that wage garnishment and other rein- 
forcement policies were based on a White, middle class belief of an absent 
father’s lack of desire to contribute to his children’s needs. This underlying 
tenet is largely not applicable to low-income fathers, who do not have the 
ability to pay (Cancian, 2020). 


Broaden your perspective through personal reflection 


Time for personal reflection is critical to processing information that may 
be new to your worldview or foreign to your discipline. After considering 
the history and context, take some time to reflect on its meaning and your 
reactions (especially visceral reactions), and ponder the implications for 
your research. Reflect on how the information affects your personal beliefs 
about people and contexts, as well as the truths that you hold (Ganly, 
2018). This is not a linear process, but a potentially complex undertaking 
that may nevertheless help you identify creative additions to your work 
and inform the recommendations that you make. While I’ve spent the last 
year reflecting deeply on race and cultural responsiveness in evaluation, 
I’ve come to the realization that I’ve always thought about it, but I’ve 
separated it from my work as it did not seem to have a place in rigorous 
economic research. I’ve now reached the point where I understand that 
the perspective I had—fostered by my graduate training and professional 
constraints—does not serve the issues that I care deeply about and hope to 
change through my contributions. 


Change an aspect (or aspects) of your research approach. 


There are a number of guides to implementing a culturally responsive 
and equitable approach to evaluation (see Public Policy Associates, n.d.; 
Hood, Hopson, & Kirkhart, 2015; Frierson et al., 2010). While many of 
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these guides are focused on evaluation, the tenets apply to primary and 
secondary research as well. Culturally responsive evaluation can be con- 
sidered at each stage in the research process (e.g., We All Count’s Data 
Equity Framework): 


e Development of research questions: You can critically consider implicit 
researcher bias, hidden assumptions, etc., that are embedded in the 
way that research questions are framed. You can also consider what 
is not being asked and whether this omission is indicative of an over- 
looked area (Thomas & Parsons, 2017). Finally, you can explore whose 
perspective is being uplifted/supported/forwarded by the research 
questions. Stakeholders have different motivations and interests, which 
do not always align (We All Count, n.d.). Since research questions are 
reflective of certain stakeholder perspectives, it is important to be 
explicit about the perspective they represent and consider whether 
the questions reflect power or influence. The review of the historical 
literature and understanding of context above can help guide the devel- 
opment of questions and investigation of what is overlooked. 


e Funding: Consider seeking multiple funders for the work, as some 
funders are focused only on rigorous quantitative analyses while others 
are willing to support qualitative research that supplements that 
approach. 


e@ Methodological approach: Most methodologies have limitations (Cody, 
2020) that can nevertheless be overcome by augmenting the work with 
additional approaches. For example, RCTs can be accompanied by 
implementation research and rich qualitative research to secure per- 
spectives from participants. In general, we can set up research projects 
that do not stop at zero impact or statistically significant findings, but 
instead go further by employing the voices and experiences of those 
directly affected to inform the answer to “why?” Our approach can 
also incorporate a greater understanding of the context, which may 
illuminate issues beyond the spotlight topic (e.g., how housing policy 
can affect educational outcomes). This may provide more nuanced and 
accessible policy recommendations, which may be more scalable and 
suffer fewer unintended consequences. 


e Data collection: Instruments and protocols for data collection need to 
reflect the stakeholders of interest and be absent of deficit language and 
stereotypes. Let’s strive to obtain multiple data sources (including qual- 
itative) and examine multiple truths (augmenting causal findings). 


e Analysis: Critically consider how variables are developed and what they 
could mean for all audiences. For example, when you have the choice to 
characterize a binary variable in the positive or the negative, why have 
you made that choice? Reporting on the proportion of African American 
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male students that receive a grade of D or lower has a different meaning 
than a report of those who received a grade of B or higher. This is the 
choice of the analyst (and may seem like a trivial choice for an outcome 
variable), but it could have profound implications for whether a 
program is interpreted as working or not. 


Reporting: Researchers have a standard way of reporting that may 
reinforce deficit framing and paternalistic stances. Consider using a 
culturally responsive or equitable evaluation checklist to review any 
publications prior to finalizing. If possible, share your report or major 
findings with stakeholders reflective of the target population to obtain 
other perspectives on framing and interpretation. 


Dissemination: Go beyond publishing in your field’s journal of choice 
and create engaging, policy-relevant one-pagers, blog posts, info- 
graphics, or other materials that help make your findings more 
accessible to a broader set of stakeholders. Think about developing 

a strategy in the early stages of your research, which will help you 
identify stakeholders and share information according to their needs. 


Take personal and professional responsibility for change 


Finally, we can all take personal and professional responsibility for indi- 
cating blind spots in our research and the research of our colleagues 
(Thomas et al., 2018). We can do this by understanding that there is a 
system within which we are operating. We can also lose the fallacy that 
research is objective and data are unbiased—they both reflect the beliefs of 
the researcher, who has a perspective and determines what, how, and why 
to examine a particular set of data. We can become more courageous in 
naming racism clearly when it exists or is a factor, rather than omitting this 
fact or using coded language about social policy (Quadango, 1994). Finally, 
we can give grace when our colleagues do not quite get it right. 
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Conclusion 


gain, research to reduce inequality rests on questions of “What 
works”? and “For whom?” But we also need to ask “How?” and 
“Why?” We need to account for the larger historical and social 
contexts in which we operate. This point is clear to many BIPOC 
who are subjects or participants in research: The odds are not fair or 
even or in your favor. As researchers, seeking to improve these odds while 
eliding historical legacies will produce only incomplete responses. 


While I have suggested that we adopt changes in the ways we conduct 
research, I am not proposing that this will be a cure-all, or that I know all 
there is to know about culturally responsive and equitable evaluation. In 
fact, Iam still learning, reflecting, applying, and reiterating in my own 
work. But while I do not have all of the answers, I have learned a lot more 
about policy and my power for change as an evaluator than I would have 
learned without deep reflection on race and equity. I truly believe that 
acknowledging and encompassing a larger view can help us move forward. 
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